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Abstract 

o 

Matrix rank minimization (RM) problems recently gained extensive attention due to numerous 
' ^ applications in machine learning, system identification and graphical models. In RM problem, one 

aims to find the matrix with the lowest rank that satisfies a set of linear constraints. The existing 
algorithms include nuclear norm minimization (NNM) and singular value thresholding. Thus far, most 
of the attention has been on i.i.d. Gaussian measurement operators. In this work, we introduce a new 
class of measurement operators, and a novel recovery algorithm, which is notably faster than NNM. 
The proposed operators are based on what we refer to as subspace expanders, which are inspired 
O by the well known expander graphs based measurement matrices in compressed sensing. We show 

that given an n x n PSD matrix of rank r, it can be uniquely recovered from a minimal sampling of 
" Oinr) measurements using the proposed structures, and the recovery algorithm can be cast as matrix 

inversion after a few initial processing steps. 

ON 

cn 

^* 1 Introduction 
O 

In the rank minimization problem, one would like to find a matrix X with the lowest rank that satisfies 

>■ a set of linear constraints A{X), often of smaller size than the number of matrix entries. This problem 

^ in its full generality is NP-hard. However, a number of recent papers have demonstrated that under 
$-1 

d certain circumstances, and when the unknown matrix is sufficiently low rank, RM can be solved via 
convex optimization programs, mainly nuclear norm minimization [U [21 [3l [15]. Random linear Gaus- 
sian measurements are considered as a standard choice, for which certain recovery guarantees can be 
asymptotically proven for NNM. The two main conditions that certify the success of NNM are Restricted 
Isometry Property (RIP) [2j and null space conditions [3j. Based on the analysis initially developed by 
Stojnic for sparse recovery, tight analytical thresholds for NNM have been recently found by Oymak et 
al. and Chandrasekaran et al. using a "escape through the mesh" analysis of the null space conditions 
when measurement operators are i.i.d. Gaussian jl5 1 116 1 [T7]. I^i parallel, there have also been promising 
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results on matrix completion problem as well, whereby one observes a subset of the entries of a low rank 
matrix, rather than linear combinations [4, 5]. 

The RM problem is often regarded as a generalization of compressed sensing(CS), wherein the aim is 
to recover a sparse vector from a set of ill-posed linear measurements, represented by a wide measurement 
matrix with m < n [T2]. It is now well understood that although certain random ensembles of 

measurement matrices (e.g. Gaussian, partial Fourier, etc.) are legitimate choices for CS, carefully 
designed matrices can lead to additional benefits for the sparse compression/recovery. A few examples 
are faster encoding time and recovery algorithms for sparse matrices and in particular expander graphs [6l 
EIEIE], higher recovery thresholds for measurement matrices based on Reed-Solomon codes |1H I14j etc. 
To the best of our knowledge, this point has not been studied in RM problem, where only random 
measurement ensembles (mostly Gaussian) have been studied. Do alternative recovery algorithms for 
RM problem exist, with success guarantees on certain classes of carefully designed linear measurement 
operators? 

In this paper, we introduce a new class of measurement operators, along with a novel recovery algo- 
rithm that is provably successful for the proposed operators, and is faster than NNM. The premise of the 
recovery algorithm relies on the measurement operator A{-) : M"^" — )• M™^™ having three key properties: 
Hermitian, low density and rank expansion. Low density property means that A{X) can be described 
as the linear combination of d linear operators A{X) = X^^^^ Ai{X), where d is a constant, and for each 

1 < i < d, the rank of Ai{X) is not larger than the rank of X . In fact, the low density operators that 
we introduce are characterized by only 0{mn) variables, as opposed to full i.i.d Gaussian linear measure- 
ments that require (mn)'^ variables. The interpretation of expansion is that A{-) maps a sufficiently low 
dimensional subspace to a higher dimensional subspace, which is equivalent to mapping every positive 
semidefinite matrix X to a positive semidefinite matrix A{X) with rank greater that c • rank(X), where 
c > 1 is a constant. The contributions of this paper are thus threefold. We first prove that low density 
rank expander operators exist. We then provide a uniqueness result, in the sense that a sufficiently low 
rank PSD matrix X is the unique PSD solution of the equations given by a high quality rank expander. 
We further propose a new recovery algorithm and provide theoretical recovery guarantees when the sug- 
gested rank expander measurement operators are exploited. Although most of the results are this paper 
are stressed for the cased of PSD matrices, we briefly mention a generalization of our results to the case of 
Hermitian matrices. This includes both the existence of high quality expanders operating on Hermitian 
matrices, and the solidity of the recovery algorithm. The validity of the proposed algorithm is verified 
by numerical simulations. 

2 Basic Definitions and Lemmas 

Let denote the space of Hermitian matrices of size nxn, and S" denote the set of positive semidefinite 
(PSD) matrices. An orthogonal projection is a matrix P G S" with = P. We say that U G IK"i^"2 
a partial unitary matrix if U^U = I, i.e. the columns of U form an orthonormal set. Notice that UU'^ 
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is an orthogonal projection. Let r]^{X),r]^{X),riQ{X) denote the number of positive, negative and zero 
eigenvalues of X, respectively. Also for a Hermitian matrix X, let X^ and denote the PSD matrices 
induced by the negative and positive eigenvalues of X respectively, i.e. X = X+ — X-. 

For a given matrix X G M"i^"2^ Xi{X) and cri{X) denotes i'th largest eigenvalue and the i'th largest 
singular value, respectively. The nuclear norm, spectral norm and Frobenius norm operators are denoted 
by II • 11^, II • II and || • \\p, respectively, and are defined by ||X||^ = ^^^^{"I'^a} (j.(^x), \\X\\ = <ti{X) and 

1 /2 

II^IU = (^Sii'i^"^'"^"'^ '^j^(^)) • addition, we define Span^(X), Span^(X) to be the linear spaces 
spanned by the columns and rows of X, respectively. 

A function / : M" — )• M is called L-Lipschitz if \f{x) — f{y)\ < L||x — yH^j, for every x,y. For hermitian 
matrices A,B, A ^ B means that A — B is positive semidefinite. For a linear operator A{-) acting on 
a linear space, we denote the null space of A by M{A), i.e. W £ M{A) iff ^(W^) = 0. We denote by 
G{di,d2) the ensemble of real di x d2 matrices in which the entries are i.i.d. A/'(0, 1) (zero-mean, unit 
variance Gaussian). 

The following lemmas are crucial to the technical discussions of this paper. Their proofs are skipped 
due to space considerations. 

Lemma 2.1. Let f{X) he a function on matrices in the following form: f{X) = 'YllLi (^i<^i{X) for some 
real constants {aj}™;^. Then f{X) is a \Jy11^i o-f Lip schitz function of X . 

Lemma 2.2. (A Gaussian concentration inequality, [20j) 

Let X be drawn from G{n, 1) and f : M" — )• M 6e a function with Lipschitz constant L. Then, we have the 
following concentration inequality 

P(|/(x) - E/(x)| > t) < 2exp(-^) (1) 

Lemma 2.3. (Weyl's Inequalities, [lOj) 

Let A,B Then: 

\j{A + B)<\i{A) + \j-^+l{B) yi<j (2) 
\j{A + B)> \i{A) + \j^,+n{B) y 3<i (3) 

3 Rank Expanders and Proposed Operators 

Definition 1. Let A : M"^" — )- he a linear operator with m < n. For d > 0,0 < e < 1,1 < r^ < n, 

we say that A is an unbalanced {e,d,rQ,n)-rank expander, if it satisfies the following conditions: 

1. For every X ^E^, A{X) G 

2. For every X G S", rank{A{X)) < d ■ rank{X) 

3. For all orthogonal projections P with rank{P) = r < rQ, rd > rank{A{P)) > (1 — e)rd 
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This definition is inspired by the definition of unbalanced expander graphs, that maintain similar 
properties with respect to positive vectors (instead of PSD matrices) and with £o-norm (instead of rank). 
An unbalanced d-regular (k, e)-expander graph is a bipartite graph with n nodes on the left and m nodes 
on the right, and regular degree d for left hand side nodes, such that every subset S of left nodes with 
\S\ < rQ has a neighborhood N{S) of size at least |A^(5')| > (1 — e)|5|d. Unbalanced expander graphs 
have been proven to have elegant properties that make them suitable for sparse vector recovery (a.k.a. 
compressed sensing) in addition to being useful as parity check matrices for error correcting codes. With 
that in mind, one might be inspired to generalize the notion of expander graphs to subspace (rank) 
expanders, in order to obtain operators that can be used in low rank matrix recovery. The following 
lemma is immediate. 

Lemma 3.1. If A{.) is an (e,d,ro,n)-rank expander then, for every X G S" with rank r < ro, we have 
rd > rank{A{X)) > (1 - e)rd 

We now move on to describe the proposed measurement structures. Afterwards, we prove that these 
constructions indeed result in rank expanders and that the expansion property allows us to find alternative 
fast reconstruction algorithms for the RM problem. 

3.1 Proposed Measurement Operator 

Let Gi, . . . G i^ni-xn |-,g matrices to be specified later. The proposed measurement operator A{-) has 
the following low density form: 

d 

A{X) = Y,G^XGf (4) 

i=l 

Where X S i?"^". We will prove that upon appropriate choices of Gj's, A{-) is an unbalanced rank 
expander. It is easy to check that with this choice of A{.), conditions 1 and 2 of Definition [T] are 
immediately satisfied. Furthermore, as long as the GiXGf's are almost incoherent, one would expect 
their ranks to add up. In particular, it is easy to show that when X is fixed and {Gi}f^i are drawn i.i.d. 
from Q{m,n), we have 

P(rank(^(X)) = min{d x rank(X), m}) = 1. (5) 

However, the challenge of condition 3 is in the fact that the rank expansion property must hold for every 
sufficiently low rank X. Let X G and X^^"^ denote an arbitrary square root of X (i.e. X = X^/^X'^/^). 
Note that A{X) can be written in the following form: 

A{X) = (GiXV2 . . . (GiXV2 . . . GdX^^Y (6) 

It then follows that 

Tank{A{X)) = rank (GiX^^'^ GaX^/^ ^ _ G^X^/^) (7) 
For analyzing the rank expansion property, we can thus limit ourselves to the form ([7]). 
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3.2 Existence of Rank Expanders 

Our goal is to prove the existence of high quahty (small e) rank expanders for certain regimes of d, e, tq 



and n. Based on Lemma 3.1, we can restrict our attention to X being an orthogonal projection of rank 
at most vq. Our analysis is for the case when Gj's are chosen i.i.d. from Q{m,n) The main existence 
theorem is the following: 

Theorem 3.1. Existence of Rank Expander For any < e < 1 there are constants Ci and C2 so 
that for any n and tq < n, whenever m = \/CiC2nro and d = ^ and {Gi}f^^ 's are independent 
instances of Q{m,n), the operator A{X) = Yli=i^i-^^f ^■^ '^'^ i^jd,rQ,n) expander with probability at 
least 1 — exp(— r2(n)). 

Before explaining the technicalities involved in the proof of the above theorem, consider the following 
argument. Given A : M"^" — R™^"*, suppose for all X G with rank(X) < r* , X can be uniquely 
decoded from A{X), say by exhaustive search. It then follows that A has to be injective on the restricted 
domain {X G : rank(X) < r*}. It will soon become apparent in the sequel that given an (e, d, ro,n) 
expander, for r* = ro/2 this condition holds. A simple argument counting the degrees of freedom of the 
low rank domain and the corresponding range of A reveals that the problem parameters should satisfy 
the following relationship. 

m = ^l.{^/nrQ), md = Vt[n)^ dr^ = 0{m) (8) 



In fact, it turns out that Theorem 3.1 is true as long as (|8|) holds asymptotically, which implies the 



optimality of the number of measurements in the suggested expander operators. For the proof, we set 



m = CidrQ and dm = where Ci > 1, C2 > 1 will be the constants in Theorem 3.1 



Proof sketch of Theorem 3. 1 . The proof is based on three major technical steps. 

Step 1: We consider an eo cover with operator norm || • || over the set of orthogonal projections of rank 
r < tq. From pTS], we know that there is such a cover of size at most M = (Co/eo)"'~, which we denote 
by {UiUf}f^-^ G S!J:, with Ui G M"^^'. Also, we first focus on a particular rank r < ro, and later union 
bound the undesirable probability over all values of r < tq. 

Step 2: Now consider a [/, from the eo cover. Denote B{Ui) = [GlU^ ... GdUi] G R^x*'. Since 
YSink{A{UiUf)) = rank{B{Ui)), we can focus on B{Ui). Note that since Ui is a fixed partial unitary, due 
to the unitary invariance of i.i.d. Gaussian matrices, B{Ui) has i.i.d. Gaussian distribution. Now define 
the function f{X) = E£(i-e)dT-+i ^i(^)' for X G R'^x'^'^. Notice that f{X) > implies rank(/(X)) > 
(1 — €)dr, because it means some of the smallest edr singular values of X are nonzero. On the other 
hand f{X) is a linear function of the singular values of X, and thus it satisfies the Lipschitz condition of 



Lemma 2.1 Since B{Ui) is Gaussian, we can apply Lemma 2.2 to get the following concentration bound 



FifiBm) < 5^) < exp(-^^^^) (9) 



^Existence of expanders using other ensembles of matrices G, and in particular sparse matrices, shall remain as an 
interesting open problem 
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where C3 = 1 — \J\JC\ and e = edrm. Here 6 serves as a safety margin, in order to account for the 
perturbation P — UiUf , when we consider a certain orthogonal projection P, which we know is the 
proximity of some UiUf of the cover, i.e. ||P — UiUf\\ < eo- In other words, as will be shown in the 
next step, lower bounding f{B{Ui)) certifies that rank(^(P)) will also be large, and thus ^ is basically 
determines a lower bound on the probability of failure. The exponent of the righthand side of ^ is 
e = edrm = eC2rn which is proportional to the exponent of the size of the cover log(M) = 0{rn). 
Consequently, with careful choices of parameters, using a union bound on failures, we can make sure 
f{B(U)) > f^;^ for all U G {Ui} w.h.p. In particular we need: 

(C3-(5)2eC2 > (logCo-logeo) (10) 

Step 3: Now it remains to show that E = P — Pi is a perturbation on Pi which makes rank(P) < 
(1 — €)dr, then \\E\\ has to be large because f{B{Ui)) is large. In particular, showing ||ii^|| > eo will finish 



the proof since we know that ||P — Pj|| < eg. In order to show this step, we make use of Lemma 2.3 to find 

^i+{l-t)dri 



1 /2 

Xi^(i-^)iiri'^iPi)) < and hence to deduce that ||;B(£', )||^ > f{B{Ui)). We carry out some more 



arguments to upper bound ||i3(£'y )||* in terms of ||£'|| to get a contradiction as long as 5^/e > ( 



holds. Finally, we conclude that whenever (*) and the condition (10) are satisfied, with high probability 
f{B{^/P)) > for all projections P with rank r, which implies rank(^(P)) > (1 — e)dr as desired. In 
particular, sufficiently large values of Ci,C2 will do the job. Also, inequalities ([s]) will similarly work 
since increasing Ci, C2 only improve the conditions. ■ 



4 Fast Recovery Algorithm 

Before presenting the main algorithm, we will provide some results about low rank positive semidefinite 
matrix recovery. Specifically, we emphasize on a uniqueness result concerning PDS matrices and the rank 
expander operators. Suppose a matrix Xq G S" and a linear operator A{-) are given, and we ask under 
what conditions Xq is the unique (PSD) inverse image of A{Xq), i.e. it is possible to recover Xq by 
simply characterizing the set {X \ X ^ 0,^(X) = ^(Xq)}. The following lemma which is adopted from 
[181 fT5] provides an answer to this question. 



Lemma 4.1. Any PSD matrix X of rank at most r is the unique PSD inverse image of A(X.), if and 
only if every nonzero Hermitian W G J\f{A) has at least r + 1 negative eigenvalues. 



Now, we explain how rank expanders can facilitate the existence of the condition in Lemma 4.1 



Lemma 4.2. Let A{.) be an {e,d,rQ,n)-rank expander with e < 1/2. Then for every nonzero Hermitian 
W G J\f{A) we have that r]^{W) > ro/2. 

Proof Any W G J\f{A), write W = W+ - Since A{W) = we have B = A{W+) = A{W-). 

Assume r]^{W) < r^jl. Let r = min{r/_ (M^), r/_|-(W^)}. Then dr > rank(i?). On the other hand 
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Algorithm 1 Reconstruct a low rank PSD matrix X from under-determined linear measurements Y — 



1: Input: 

2: Constant integer d> 1. 

3: Matrices Ai G M™^", 1 < i < d, and F G M"^™. 
4: Output: 

5: Low rank PSD matrix X. 
6: Initialize 

7: Compute Y = ST.S*, with S full column rank (SVD). 
8: Set P = I - SS*. 

9: Set Q := Nun((P^i)^, . . . , {PAa)^)^. 
10: Compute Bi = AiQ, and set M = Yfi=i ^ Si- 
ll: Find X G W"""- with vec(X) = (Q Q)Aftt>ec(y). 



rank(VF+ + = rank(W+) + rank(W_) > 2r hence we have rank(^(py+ + W-)) > 2(1 — e)dr since 
2r < ro. Note that A{W+ + = A{W+) + A{W-) = 2B. It follows that dr > rank(5) = rank(25) > 
2(1 — e)fir =^ 1 > 2(1 — e) <^=^ e > 1/2, which is a contradiction. ■ 



The combination of Lemmas 4.1 and 4.2 suggests that by using an (e, d, ro, n)-rank expander with 
e < 1/2 as a measurement operator, one can guarantee that every PSD matrix Xq of rank at most 
ro/2 is the unique PSD solution to the measurements A{Xq). Therefore, every program (e.g. SDP) 
that can identify a point in the feasible set {X \ X ^ 0,A{X) = ^(Xq)} successfully returns Xq. 
Quite Interestingly, for the case of high quality expanders (small e), we propose an alternative method 
for identifying one feasible point, which by the token of the aforementioned uniqueness argument, can 
successfully recover low rank PSD matrices. The algorithm is based on using the expansion property 
of the suggested linear operator, and the fact that the original matrix is low rank. The key point is 
that for rank expanders, the original under-determined system can be equivalently transformed to an 
over-determined linear system, after a few simple processing steps, that mostly involves taking SVD and 
finding null spaces. The new system of linear equation can then be solved by matrix inversion. This 
routine is described in Algorithm [T| and is in fact the generalization of a positive sparse vector recovery 
algorithm elaborated in [S]. The next theorem provides a guarantee for the success of the proposed 
algorithm. 

Theorem 4.1. PSD Recovery. // the operator A{X) = Ya=i AXAJ is a {e,d,rQ,n)-rank expander 
with e < 1/2, then for every k < ro(l — e), every PSD matrix X of rank k can be perfectly recovered from 
A{X) using Algorithm^ 



In order to prove Theorem 4.1, we first prove the following lemma. 

Lemma 4.3. Suppose that the operator A{X) = Yli=i ^i^^f cl^ unbalanced (e, d, ro, n)-rank expander 
with k/(l — e) < m < n, where Ai 's are m x n. Also suppose that A G S" has rank k < ro(l — e)/2. 
Further, let S be the linear space of all nx 1 vectors u such that Span(j{A{uu'^)} G Span(j{A{X)} . Then 
dim{S) < k/{l - e). 
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Proof. If dim(5) > k/{l — e), then we can find an orthonormal matrix U of size n x k/{l — e) such 
that all of its columns are in S. Therefore, by definition SpanQ{A{UU^)} G Span(j{A{X)} , and thus 
rank(^(C/C/^)) < rank(^(X)) < k-d. However, since k/{l — e) < ro, from the definition of rank-expander, 
we must have rank{A{UU'^)) > d{l — e)rank(C/C/'^) = k ■ d, which is a contradiction. ■ 



Proof of Theorem 4-1' Let S be as defined in Lemma 4.3 with dim(5) = r, and Q"^*" be a basis for 5, 
and let X = X^^'^X'^^'^ . It is easy to check that the columns of X^^'^ are ah in S, and thus X^/^ = 
for some unknown V^^"^ of size r x k. Therefore, we can write: 

Y = A{X) = A.QV'/^V^/^Q^Aj (11) 



i=l 



or equivalently 



vec(y) = [ ^ Si vec{V) 



(12) 



where Bi = AiQ. (12) is a linear system of equations with m? equations and unknowns. Moreover, 



from Lemma 4.3, we know that r < k/ {1 — e) < m. In addition, from Lemma 4.2, the solution to the 



system of linear equations in (12) is unique and must heV = X, since otherwise X — 1/ is in the null 



space of ^(•), and has at most r < ro/2 negative eigenvalues. Therefore, (12) is an over-determined linear 
system and can be solved by matrix inversion as given in line 11 of Algorithm [T} ■ 



5 Extension to Hermitians 

In this section, we briefly extend the results of the previous sections to the case of Hermitian matrices. 
Specifically, we prove the existence of expanders for Hermitian matrices (rather than only PSD matrices, 
which was the case discussed previously) , and then state a theorem certifying the success of Algorithm [l] 
for these classes of low rank matrices. 

5.1 Expansion 

Lemma 5.1. Expansion for Hermitians. Assume A is a (e, d, ro, n) expander and let X he Hermitian 
with rank{X) < tq. Then, rank{A{X)) > (1 — 4e)(i • rank{X). 

The formal proof of the above lemma is skipped, but it is mostly based on some dimension counting 
arguments and the fact that rank(^ + i3) = rank(^) +rank(i?) is equivalent to Span(^(^)nSpan(^(i?) = 
and Span^(^) n Spanj^(5) = (see e.g. [19]). 

5.2 Recovery 

Assume X G S with rank(X) = r < tq. With Gaussian measurements, rank(^(X)) = rd almost surely, 
due to ^ and the result of [19]. Notice that if the eigenvalue decomposition of X is X = XiUiuf, 
then A{X) = Aj ^(ujU^), and using rank(^(X)) = r x d and i[ank{A{uiuf)) < d, it follows that 
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Span(j{A{uiuJ)) C SpanQ{A{X)) for all i < r. Consequently, similar to the case of PSD matrices, 
we need to find the space of u such that SpanQ{A{uu^)) C Span(^{A{X)). All technical steps follow 
identically and similar to the PDS case, and we can assert that sufficiently low rank Hermitian solutions 



to (12) are unique, as follows. 

Theorem 5.1. Hermitian Recovery. Let Xq £ S with rank{XQ) < ro(l — e). Suppose A{-) is as 
described in Theorem \3.1\ With probability at least 1 — exp{—Q{n)) , Xq can be perfectly recovered from 
A{Xo) by Algorithm\^ 

Note that although the same algorithm works for both cases of PSD and Hermitian matrices, there is a 



significant difference. In Theorem 4.1 "all" X G S" with sufficiently rank low are recoverable, whereas a 
similar fact is true for "almost all" Hermitian matrices of low rank. These notions are often distinguished 
in the literature by the terms "strong" recovery and "weak" recovery, respectively. Furthermore, note 
that for the case of PSD recovery, one can alternatively use convex optimization to find the unique PSD 
solution. However for the recovery of Hermitians, we do not know of any other method but our proposed 
Algorithm [TJ 

6 Simulation Results 

Numerical simulations were performed to verify the validity of Alg. [TJ We used n = 50, and linear 
measurement operators in the form of Q with d = 2,3, 4, and two types of distributions for Gj's: 1) i.i.d 
Gaussian matrices, and 2) sparse matrices where every row of each d has exactly one 1 in a random 
location. We did not explicitly prove that the sparse constructions expanders. However, such low density 
structures are of high practical interests. The resulting curves of successful recovery thresholds are given 
in Figure [T] In all of our simulations for Gaussian matrices, the transition between successful recvery and 
failure was very sharp, i.e. either failed all times or succeeded, depending on the number of measurements 
and the rank of X. Figure [T] illustrates the empirical transition phase or the recovery threshold of Alg. 
[TJ The corresponding curves for sparse matrices are also shown in Figure [T} The transition for sparse 
matrices was not as sharp as Gaussians. On the same curves, the performance of the standard trace 
minimization with nonnegativity (PSD) constraint is also displayed. Observe that the performance of 
the proposed algorithm is very comparable to the convex relaxation method. In addition, the curves for 
sparse measurements collapse into the same curves as the dense measurements for sufficiently large m. In 
practice, Alg. [T]is extremely faster than NNM. To give an example, for simulations in MATLAB on a 2.4 
GHz Intel Core 15 with 4 GB RAM, and for the case of d = 2,m = 39, k = 10 with Gaussian Gj's, each 
reconstruction took on average 34s for the NNN, while it was only about 0.05s for Alg. [T] For sparse 
structures, it took about 1.8s for NNM and 0.05s for Alg{l} NNM was solved via the SEDUMI toolbox. 
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